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ABSTRACT 



This study evaluates the usability of a U.S. Navy 
Decision Support System (DSS) . The DSS was developed to 
enhance the performance of tactical decision-makers 
within a Navy Combat Information Center. The goals of 
this study were to test the DSS against usability 
criteria and objectives to track future redesign efforts 
and system improvements. The purpose of this analysis 
was to (1) assess the system's usability, (2) identify 
problems areas in the graphical user interface, (3) 
report trends in user feedback, and (4) provide 
recommendations addressing major usability issues 
encountered by participants. The study tested whether 
the DSS met the usability objectives of (a) 90% 
successful task completion, (b) ease-of-use ratings of 
somewhat easy or better, and (c) satisfaction ratings of 
somewhat satisfied or better. The DSS did not meet these 
usability objectives for task completion or ease-or-use, 
however the DSS did meet the usability objective for 
user satisfaction. All participants reported that they 
enjoyed working with the DSS and believed that it would 
be a significant step forward in information management. 
Based on the usability data gathered in the study, 
recommendations are provided to address the usability 
issues . 
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EXECUTIVE SUMMARY 



This study evaluates the usability of a U.S. Navy 
Decision Support System (DSS) . The DSS was developed to 
enhance the performance of tactical decision-makers 
within a Navy Combat Information Center. The DSS is 
still in the development phase and has continually been 
improved based on empirical studies and subject matter 
expertise. The most recent prototype version, known as 
the DSS-2, is the focus of this study. The goals of this 
study were to test the DSS-2 against usability criteria 
and objectives to track future redesign efforts and 
system improvements. The purpose of this analysis was to 
(1) assess the system’s usability, (2) identify problems 
areas in the graphical user interface, (3) report trends 
in user feedback, and (4) provide recommendations 
addressing major usability issues encountered by 
participants. The study tested whether the DSS met the 
usability objectives of (a) 90% successful task 
completion, (b) ease-of-use ratings of somewhat easy or 
better, and (c) satisfaction ratings of somewhat 
satisfied or better. The DSS-2 did not meet these 
usability objectives for task completion or ease-or-use, 
however the DSS-2 did meet the usability objective for 



XV 



user satisfaction. All participants reported that they 
enjoyed working with the DSS-2 and believed that it 
would be a significant step forward in CIC information 
management. Based on the usability data gathered in the 
study, recommendations are provided to address the 
usability issues. 

The methodology applied in this study was useful in 
the evaluation of the DSS-2. This study demonstrated 
that traditional human-computer interface usability 
methods could be directly applied the evaluation of 
synthetic environments. The DSS-2 is a simple synthetic 
environment represented on two computer monitors . Given 
the success of this methodology with the DSS-2, it would 
be appropriate to use this methodology in evaluating 
more complex synthetic environments. 



XVI 



ACKNOWLEDGEIXIENTS 



The author would like to express acknowledgement 
and appreciation to my thesis committee members. Dr. 
Rudolph Darken and Captain George Conner, USN (retired) 
for their assistance, direction, and dedication 
throughout my course of study. Acknowledgement and 
appreciation is also due to Ms. Monica Heidelberg for 
her direction in the development of the usability 
methodology employed and in the analysis of the data. 
Gratitude is expressed to Dr. Jeffery Morrison for 
providing the resources to accomplish this thesis. 

Appreciation is expressed also to Dr. Michael Zyda 
for his support and assistance. Gratitude is due to Dr. 
Maurice Weir for his guidance on obtaining this degree. 

Acknowledgment and gratitude is expressed to Monica 
Heidelberg for her love, support, sacrifice, and 
encouragement needed to bring this study to completion. 
Appreciation and gratitude is expressed also to my 
parents, Linda Nowland and David Schmorrow; my 
grandparents, Louis and Mary Schmorrow; and the rest of 
my family for their patience and encouragement 
throughout the duration of my studies . 



XVll 



xviii 



I. 



INTRODUCTION 



A. OVERVIEW 

The United States Navy and Marine Corps strategy of 
forward presence suggests that they will be first on the 
scene in times of crisis. Furthermore, since a majority 
of the world's population lives within 200 miles of the 
ocean, most future contingencies are likely to involve 
littoral warfare (Mundy, 1994) . This strategy will 
increasingly place Naval forces in coastal areas where 
they will be forced to operate in confined and congested 
areas (Hutchins, Kelly, & Morrison, 1997) . These 
constraints will result in operational scenarios that 
will require both increased information processing and 
accelerated decision-making. These challenges are 
exacerbated by the fact that current real-time battle 
management systems are primarily effective in dealing 
with all-out conflicts and not particularly capable in 
situations, such as littoral conflicts, where human 
intervention in decision-making is more critical 
(Hutchins, Morrison, and Kelley, 1996). 

Two unfortunate and well-known incidents involving 
the U.S.S. Stark and the U.S.S. Vincennes highlight this 
challenge. The U.S.S Stark incident centers on a 
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decision made by the commander not to engage an inbound 



aircraft. The 


aircraft 


was 


not 


considered 


a 


threat by 


the commander. 


however 


it 


was 


a threat. 


The 


aircraft 


significantly 


damaged 


the 


U. S 


. S. Stark 


and 


numerous 



lives on board were lost. The commander of the U.S.S. 
Vincennes faced a similar problem, yet believed the 
inbound aircraft he faced was a threat to his ship. As a 
result, the inbound aircraft was destroyed by the U.S.S. 
Vincennes. The aircraft turned out to be a commercial 
passenger airline and all lives onboard the airliner 
were lost. The Tactical Decision-Making Under Stress 
(TADMUS) program was initiated to address these types of 
problems. The principle product of the TADMUS program is 
the Decision Support System (DSS) . Due to the 
criticality of these issues and the need to correctly 
identify threats, it is imperative that the design of 
the TADMUS DSS system be intuitive and easy to use. This 
study will evaluate the usability of the TADMUS DSS 
graphical user interface . 

B. BACKGROUND 

The DSS system was developed to enhance the 
performance of tactical decision-makers. It was derived 
from current cognitive theory. This derivation first 
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analyzed the cognitive tasks performed by decision 
makers in a shipboard Combat Information Center and 
second, developed a set of displays to support these 
tasks based on the underlying decision making processes 
(Morrison et . al . , 1997) . The DSS is currently a 
prototype and is planned to be formally tested onboard a 
Navy ship in 1999, see Figure 1. 




The DSS is still in the development phase and has 
continually been improved based on empirical studies and 
subject matter expertise. The most recent prototype 
version, known as the DSS-2, is the focus of this study. 
A usability test was conducted on the DSS system to 
evaluate human performance and user preferences. This 
test also identifies usability issues that focus on 
future design and redesign efforts. 
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C. PROBLEM STATEMENT 



Given the critical nature of the tasks supported by 
the DSS system, and the implications of a difficult to 
use design, a usability study was conducted on the DSS. 
The goals of this study were: 

• Conduct a usability study to test where the DSS-2 
product is today in regards to usability 
criteria . 

• Track usability measures (e.g., successful 
completion of tasks, error rate, time to complete 
task, ease-of-use, and user satisfaction) in 
order to track product improvement . 

• Identify usability issues to address future 
design iterations. 

• Provide user feedback to DSS development team. 

• Provide recommendations to address usability 
issues encountered by users during testing. 

D. OBJECTIVE 

The purpose of this study is to assess the 
usability of the human-computer interface of the TADMUS 
DSS. The long-term objective is to provide a methodology 
and baseline information for the evaluation of future 
systems . 
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E. SCOPE AND LIMITATIONS 



To narrow the scope of the thesis, only human- 
computer interaction performance and preference will be 
analyzed. Issues concerning conceptual cognitive 
decision-making will not be addressed. 
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II. LITERATURE REVIEW 



A. OVERVIEW 

The review of literature for this research included 
journals and textbooks covering the subjects of 
usability evaluation, human -computer interaction, and 
synthetic environments. The purpose of this literature 
review is to provide an overview of the historic and 
current theories and practices relating to usability 
evaluation and to provide information on the methods 
used in this study to evaluate the DSS. 

B . BACKGROUND 

There has been significant growth in the fields of 
synthetic environments and usability engineering. The 
term synthetic environment is used to refer to virtual 
reality, virtual environments, teleoperator system, 
telerobotic systems, augmented reality and synthetic 
environments in general. However, these two fields have 
experienced growth independently. "An underlying 
assumption among both (synthetic environments) 
researchers and developers sometimes seems to be that 
(synthetic environments), because they are a novel and 
impressive technology, are inherently good and usable 
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(Gabbard and Hix, 1997, p.3) The tools developed and 
the lessons learned in the field of usability 
engineering have yet to be significantly applied to 
synthetic environments and those that have been applied 
typically have not addressed the broad issues of 
usability throughout the system (National Research 
Council, 1997; and Gabbard and Hix, 1997). The 
integration of these two fields will mutually benefit 
both. Usability engineering will gain a technologically 
savvy customer and developers of synthetic environments 
will drastically improve the usability of their 
technologically complex, and frequently difficult to use 
systems . 

Usability engineers will need to modify existing 
methods and tools as well as develop new ones 
specifically for synthetic environments. For example, 
typical human-computer interaction usability studies 
focus on standard graphical user interfaces where there 
is a single user. In the synthetic environment, 
innovative and non-standard methods of interaction as 
well as a multi-user capability call for a redefinition 
of the current usability paradigm. To facilitate this 
transition, Gabbard and Hix (1997) have outlined the 
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four primary usability characteristics 



related to 



synthetic environments. These are: 

1 . Users and User Tasks in Synthetic Environments - 
general user and task characteristics and types 
of tasks in synthetic environments. 

2 . The Virtual Model - usability characteristics of 

generic components typically found in synthetic 
environments . 

3 . Synthetic Environment User Interface Input 

Mechanisms - usability characterist ics of 

synthetic environment input devices. 

4 . Synthetic Environment User Interface Presentation 
Components - usability characteristics of 

synthetic environment output devices . 

Gabbard and Hix have developed a comprehensive 
taxonomy based on these four areas in order to move 
beyond the "let's build it and see what happens" method 
that is often employed in synthetic environments. This 
taxonomy is a classification, enumeration, and 

discussion of usability issues in synthetic environments 
and was developed to ensure that usability will be 
integrated into the development of synthetic 
environments. In addition to this work, additional 
analysis of usability engineering and its integration 
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into synthetic environments through usability testing 
needs to be undertaken. The first step in determining 
how this can be accomplished is to examine usability, 
usability engineering, and synthetic environments. 

C. USABILITY 

Usability engineering is a systematic approach to 
usability. In general, usability means that the people 
who use the product are able to do so quickly and easily 
to accomplish their own tasks (Dumas and Redish, 1994) . 
This definition is based on four €>ssential points, they 
are ; 

1 . Focus on users . 

2. People utilize products to be productive. 

3. People have limited time to accomplish tasks. 

4. Users decide when a product is easy to use. 
Usability is concerned with the sum total of a product - 
Usability should not only be considered an issue for the 
primary system functionality, but should also be applied 
to training materials, help packages, and other 
associated features of the system. In order to improve 
the ease-of-use of a product, usability should be 
considered throughout the development of a system, from 
initial design through final deployment of the system. 
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Dumas and Redish (1994) provide seven principles 
for ensuring usability: 

1. Engineering it into a product through an iterative 
design and development process. 

2 . Involving users throughout the process . 

3. Allowing usability and users' needs to drive design 
decisions . 

4 . Working in teams that include skilled usability 
specialists, interface designers, and technical 
communicators . 

5. Setting quantitative usability goals early in the 
process . 

6. Testing products for usability, but also integrating 
usability testing with other methods for ensuring 
usability. 

7 . Being committed to making technology work for 
people . 

This integration of usability into a product is commonly 
called usability engineering, (Good, 1988; Whiteside, 
Bennett, and Holtzblatt, 1987) . Similar to software 
engineering, usability engineering includes identifying 
users, analyzing tasks, setting specifications, 
developing and testing prototypes, and the iterative 
cycles of development and testing (Dumas and Redish, 
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1994). Gould and Lewis (1985) highlight four principles 
to facilitate designing usability into products. 

1 . Focus early and continuously on users. 

2. Integrate consideration of all aspects of 
usability . 

3 . Test versions with users early and continuously. 

4. Iterate the design. 

Identifying usability requirements prior to design can 
save time and money for the designer as well as increase 
the likelihood of user satisfaction with the product. 
Systems are developed to help individuals accomplish a 
task. In order to provide a usable system, what the 
individual needs and how they are to accomplish this 
must be ascertained. The primary requirement is to 
understand the prospective users and the audience for a 
system. Dumas and Redish (1994) have identified 
techniques that can be used in a usability engineering 
process. These techniques highlight the importance of 
describing what a person does in their job in terms of 
tasks. When the tasks are analyzed, how the person does 
the job, can do the job, or should do the job are 
described (Drury, Paramore, Van Cott, Grey, and Corlett, 
1987) . 
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Table 1: Usability Engineering Process Techniques 

Techniques for Planning, Implementing, and Evaluating Usability 

Uncovering usability needs before you design 
Identifying users' jobs and tasks 
Convening focus groups 

Interviewing and observing users in context 
Conducting usability tests of existing versions 
Conducting usability tests of competitors' products 
Setting quantitative usability goals 
Basing designs on expertise in human- computer interaction (HCI) 
Understanding the HCI and document design approach 
Using HCI and document design principles and guidelines 
Setting and using local rules 
Evaluating usability throughout design and development 
Getting experts to review the design 
Having peers or experts walk through the design 
Having users work with static prototypes 
Having users work with interactive prototypes 
Getting user edits on early versions of documentation 
Conducting iterative usability tests 

Asking users about their satisfaction 

Redish and Dumas (1994) 



In addition to understanding the principles of 
usability and the usability engineering process, it is 
important to set quantifiable usability goals early in 
the design process. By setting quantifiable goals, a 
product development team will have a concrete way to 
measure usability success. A series of quantitative 
goals with related objectives should be identified prior 
to system development and will facilitate subsequent 
analysis. A team may have a goal to design a product to 
be easy to learn and operate, however this is not a 
quantitative goal and would be difficult to measure. The 
design team needs to define quantitative goals to more 
easily measure usability. Subjective criteria can also 
be defined to help evaluate a products' usability. 



13 



Subjective criteria are often easily derived, yet it can 
be difficult to determine if the criteria have been met. 
Typically, it is easier to determine when objective 
goals have been met. However, they may originally be 
more difficult than the subjective criteria to develop. 
For instance, in an air traffic-control synthetic 
environment, a usability goal might be that users should 
be able to detect and identify a new track in less than 
5 seconds. Measures such as these are important in that 
they provide a basis for evaluation on whether the goal 
was achieved, they allow systems to be compared, and 
provide baseline information against which revisions can 
be evaluated. 

D. USER INTERFACE DESIGN 

The usability of a product is inherently tied to 
the user interface. If the user interface is intuitive, 
easy to learn and use, a product will have favorable 
usability ratings. Guidelines and user interface 
heuristics have been established by academia and 
industry experts to best design user interfaces for 
usability. Shneiderman (1997) proposes eight golden 
rules of interface design to best maximize the usability 
of an interface. These include: 
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1. strive for consistency 

2 . enable frequent users to use shortcuts 

3 . offer informative feedback 

4. design dialogs to yield closure 

5. offer error prevention and simple error handling 

6 . permit easy reversal of actions 

7 . support internal locus of control 

8. reduce short-term memory load 

If followed, these rules should foster a sense of 
comprehension and competence among users. This is 

particularly important because users prefer systems with 
which they feel familiar and competent. Furthermore, if 
a user has positive feelings toward a system they are 
more likely to highly rate the performance of these 
systems. These rules were originally developed primarily 
for the standard graphical user interface. However, 
these general underlying principles of interface design 
can be interpreted, refined, and extended to synthetic 
environments . 

Striving for consistency can be problematic in that 
consistency can relate to many aspects of the system 
(i.e., terminology, color, layout, input and display 
formats) . For example, consistency in a virtual 
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walkthrough of a house could refer to consistency of the 
visual representation of objects in the environment or 
could refer to consistency of human interaction with 
these features. It is not always possible to maintain 
consistency across all dimensions of a system, but 
identical symbology and methods of interaction should be 
employed throughout. 

Shortcuts enable frequent users to reduce the 
number of interactions required to obtain a desired 
result and also increase the pace of interaction. In the 
synthetic environment, unique methods of input and 
display need to be improved to take better advantage of 
shortcuts. For instance, there may be times when a 
three-dimensional virtual environment could be switched 
to a two-dimensional map on which the user could 
navigate. When the user reaches a desired location, a 
return to the three-dimensional world could be 
initiated. Other shortcuts could include gestures or a 
series of gestures, which are not directly relevant in 
the current environment, but offer a shortcut to another 
environment . 

Offering information feedback facilitates the 
user's immersion in synthetic environments. This 
feedback can vary in degree with infrequent and minor 
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actions resulting in small changes in the synthetic 
environment, whereas critical and major actions result 
in substantial changes in the visual presentation. 
Without substantial information feedback users may not 
be able to fully complete actions or understand their 
current status. For example, in an air-traffic control 
system the selection of an aircraft should be indicated 
through a state change (i.e., the display alters and an 
object becomes highlighted). 

Usability of a system can be further maximized by 
designing dialogs to yield closure. This can be achieved 
by grouping a set of actions to provide a natural flow 
through a users' tasks. This sequencing of actions 
provides the user better awareness of the actions taken 
and gives the user a sense of closure of the sequence. 
An example of this concept is virtual kitchens where a 
user can pick up a dish, manipulate the dish, and break 
the dish. If correctly employed, the user will clearly 
know the status of the dish and the associated action. 
At the conclusion of the sequence of actions, the user 
will clearly see the dish replaced on a counter or 
broken into several pieces and be rewarded through this 
sense of closure and awareness . 
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Whenever practical, users should be permitted to 
reverse actions if they choose. Users tend to make 
mistakes, therefore a system should be designed to allow 
users to recover from errors easily. This may reduce any 
stress or anxiety the user has when operating within a 
synthetic environment. In the virtual kitchen example 
mentioned above, a user may have selected the "wrong" 
dish. The user should be able to recover from this error 
and replace the dish in the same place as before. 
However, the virtual kitchen example also suggests when 
this would potentially not be allowed. For example, if 
the user has broken a dish, there is no recreation of 
the dish. Similarly, if in a missile fire-control 
station a missile is accidentally fired there is no 
ability to call it home. 

The design of a system should also support a users' 
"internal locus of control." Users should be the 
initiators of actions not the responders to actions 
(Gaines, 1981). When appropriate, a synthetic 
environment should be designed with the users in 
command. Whereas the status of objects in the 
environment would be appropriately updated and 
maintained without user action, the autonomous movement 
of the user within the environment or a drastic altering 



18 



of the visual orientation would be inappropriate. The 
issue of system interruptions arises in this context 
(McFarlane, 1998) . If a user is engaged in the synthetic 
environment, when is it appropriate for the user to be 
interrupted with a competing task? How should that 
interruption manifest itself? This issue of 
interruptions needs further study, in particular within 
the context of virtual environments . 

The reduction of short-term memory load is 
essential for optimal integration of a user within a 
synthetic environment. Humans are limited in their 
ability to maintain excessive amounts of information in 
their short-term memory. Designs of synthetic 
environments should include cues, mnemonics, and 
standardized sequences of actions. Whenever possible, 
access to integrated assistance information should be 
provided. For instance, in an air traffic-control 
environment, the history of the air tracks should be 
made available to the user. If a task requires a series 
of actions, a list of those actions should be available. 
When designing systems to reduce short-term memory load, 
the designer should remember that humans have been shown 
to be able to remember seven items plus or minus two. 
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These rules of interface design, which are based 
upon existing usability models, can be modified for the 
synthetic environment. However, the limitations of 
existing usability models should be understood. 
Synthetic environments have many unique characteristics 
and understanding these is essential in addressing the 
overall usability. Unique characteristics of synthetic 
environments not supported in existing usability models 
include perceived presence and perceived real world 
fidelity and existing models do not support 
quantification or qualification of a user's perception 
of such characteristics (Gabbard and Hix, 1997) . 
Traditional usability models are also limited in scope 
in that they typically focus on a single user at a 
single site. Other issues involve the multiple and 
unique methods of interaction and display that are 
continually being developed for synthetic environments. 

E. USABILITY TESTING 

A usability test primarily measures ease-of-use. 
According to Dumas and Redish (1994), "usability testing 
is a systematic way of observing actual users trying out 
a product and collecting information about the specific 
ways in which the product is easy or difficult for them 



20 



to use (p. 12)." Dumas and Redish (1994) also identify 
three basic tenets of usability testing. The first is 
that usability testing should be used to diagnose 
problems and not to determine that the product is 
flawless. The second is that usability testing should be 
employed early in the development of a product and 
often. Lastly, that usability testing is part of a 
process that focuses on usability throughout design and 
development . 

In order to best incorporate usability into the 
development process, a thorough testing plan needs to be 
developed. There are several determinants that need to 
be addressed in developing an evaluation plan 
( Shneiderman, 1997; Nielsen, 1993; Hix and Hartson, 
1993; Preece et al . , 1994; Newman and Lamming, 1995) . A 
foundational determinant is the current stage of the 
design. The requirements for testing an early design as 
compared to a late design will differ in that general 
concepts of user interaction with the design need to be 
tested early, whereas testing of a late design may be 
targeted more at identifying consistency within the 
environment and task completion. In addition, the 
criticality of the environment is a significant 
determinant in deciding the objectives of the test. The 
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level of task completion rates and number of errors 
allowed in a test will vary depending on whether an 
environment is being developed as part of a life- 
critical system or as an entertainment system. Finally, 
factors such as the novelty of the project, the number 
of expected users, the time available, the costs of the 
product, the available resources (i.e., time and money 
available for testing) , and the experience of the 
usability testers themselves play a role in shaping the 
usability test. 

Usability testing of a system has become essential 
not only to maximize the usability of the system, but 
also to verify that contractual requirements have been 
met and to document that testing has been conducted in 
case legal issues or lawsuits arise concerning the 
operating of the system ( Shneiderman , 1997) . This is 
best illustrated when one considers that perfection is 
not possible in any system, particularly systems that 
incorporate human users. The varying degrees for which 
errors will be tolerated relates directly to the 
requirements to bring the system to full operational use 
and the impact that the errors may have during 
operational use. However as Shneiderman (1997) suggests, 
systems which require high levels of input such as 
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nuclear-reactor-control or air-traf f ic-control 
emergencies are very difficult to test. However, testing 
methods to deal with stressful situations, which include 
life-critical applications, are increasingly needed. 

Another usability testing method employed to 
improve a product's usability is an expert evaluation of 
the system. Nielsen and Mack (1994) argue that formal 
expert reviews can generally provide more useful 
information as compared to informal demonstrations to 
colleagues or customers. This requires that expert 
reviewers are available to the usability testing team. 
If available, expert reviewers can be employed 
throughout the design and testing of a system. The 
typical product of an expert review is a report 
outlining identified problems and recommendations for 
improvement. The forms these reviews may take include 
heuristic evaluation, guideline review, consistency 
inspection, cognitive-walkthrough, and formal usability 
inspection. Expert-reviews do face challenges. For 
instance, expert-reviewers may be confronted with new 
systems and technology they are not completely familiar 
with and for which they may not fully understand the 
design rationale or development history. However, expert 
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reviews typically provide a fresh look at a system and 



are useful in evaluating system development. 



Table 2 : Expert Review Methodology 



Eaqpert -Review 
Method 

Heuristic 

evaluation 



Guidelines 

review 

Consistency 

inspection 



Cognitive- 

walkthrough 



Formal 

usability 

inspection 



Description 



Expert-reviewers critique an interface to 
determine conformance with a short list of 
design heuristics such as the eight golden 
rules . 

The interface is checked for conformance 
with the organizational or other guidelines 
document 

Experts verify consistency across a family 
of interfaces, checking for consistency of 
terminology, color, layout, input and 
output formats, within the interfaces as 
well as in the training materials and 
online help 

Experts simulate users walking through the 
interface to carry out typical tasks . 
Simulating the day in the life of the user 
should be part of the evaluation. 

Experts hold courtroom-style meeting, with 
a moderator to judge, to present the 
interface and to discuss its merits and 
weakness . 



Shneiderman, 1997 



Formal usability testing in laboratories can 
provide information concerning user needs and abilities 
that an expert-review may miss. Usability testing and 
usability laboratories have been developed to capture 
the user experience directly. The information gained is 
used to confirm progress in the design of a system and 
to obtain recommendations to improve upon the system. 
Typically, a formal usability study is conducted in a 
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usability laboratory in a controlled setting with a set 
of tasks for the user to undertake. 

Usability studies do take other forms, such as 
Nielsen's (1993) discount usability engineering which 
are "quick and dirty" approaches to task analysis, 
prototyping, and testing ( Shneiderman, 1997). Field 
studies are another type of usability study which are 
conducted in actual work environments in order to 
achieve realistic, user evaluation. A different approach 
to these traditional methods is to challenge actual 
users of the system to try to break the system, commonly 
called beta testing. By offering rewards to individuals 
who find flaws in a system, developers can speed up the 
development process and correct errors that may have 
been missed through conventional testing. Two serious 
flaws with usability testing in general are that it 
emphasizes first-time usage and lacks a comprehensive 
evaluation of the system due to time constraints 
(Shneiderman, 1997). These flaws necessitate that 
usability testing be supplemented with other methods of 
evaluation such as expert-review. 
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F. PLANNING FOR USABILITY TESTING 



When planning a usability test, often the most 
important question, besides what is required, is how 
long the test should take. If the usability testing is 
an integrated part of the design process and is not 
simply being conducted on a completed system, then the 
test needs to be as short as possible to obtain the 
necessary information - and short enough so that the 
test is not burdensome. This will facilitate the 
iterative nature of proper usability testing. Testing 
length depends on many factors, including how much prior 
testing has taken place, how complex the system is, and 
the scope of the system to be tested. Dumas and Redish 
(1994) suggest that traditional testing lengths fall 
into one of four categories. Organizations that follow 
formal testing and generate comprehensive test reports 
allow eight to twelve weeks. Shortened testing periods 
of four to six weeks are frequently used when there 
exists a strong collaboration between team members and a 
shortened formal report is used. When only a particular 
aspect of a system is to be studied with well- 
established procedures, one week can suffice. Just-in- 
time testing is discouraged, but can still provide 
useful information in a couple of days if necessary. 
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Studies conducted by experienced and dedicated 
individuals who take the necessary time will most often 
achieve the best results. Furthermore, proper planning 
entails the definition of goals and concerns, deciding 
who should participate and recruiting these individuals, 
developing and organizing tasks and task scenarios, 
deciding on usability measures, preparing the test 
materials and test environment, and conducting a pilot 
test . 

Dumas and Reddish (1994) suggest that defining 

goals and concerns for usability testing can be viewed 

as a three-stage process. The first stage is making 

choices among goals and concerns. For instance. 

Is your main concern whether new users 
will be able to get up and running to 
do basic tasks quickly, or whether 
users who have had the product for 6 
months can figure out more advanced 
functions? You may be concerned about 
both, but you'll have to plan two 
different tests to learn about both 
(Dumas and Redish, 1994, p. Ill) . 

The second stage is moving from general concerns to 

specific ones. This helps determine the type of subjects 

necessary and begins to shape concerns into quantitative 

objectives. Lastly, understanding the sources of these 

goals and concerns allows the usability engineer to 

better develop the testing scenarios and tasks . Some 
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sources include expert-reviews, user feedback, and 
previous tests. 

The decision on who should participate in the study 
should be based on developed user profiles. Ideally, 
user profiles should have been developed prior to design 
of the system and usability testing. If this has not 
been done, a user profile can be developed by 
identifying all the relevant characteristics that an 
individual using the system should have. The two primary 
characteristics of concern are those that all the 
individuals have in common and those that may make a 
difference between the individuals. For example, if a 
command and control synthetic environment were being 
deployed onboard a US Navy ship for the first time, 
certain questions need to be addressed, these include: 

1. Will many users be working with abstract or 
simulated environments for the first time? 

2. Will many individuals be experienced with 
personal computer applications, but new to the 
synthetic environment? 

3 . Will many users already be adept at using the 
input devices? 

4 . Who will be using this system -- commanding 
officers, junior officers, or enlisted personnel? 
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The decision on how many subjects are required for 
a study can also be a challenging question. Usability 
engineers are often delighted to have ten to twelve 
subjects participate in a study, whereas a statistician 
might insist on no less than thirty-six to forty-eight. 
The realities of time and budget constraints often 
result in usability studies having six to eight 
subjects. A simple answer to the question of how many 
subjects to use is enough participants to complete the 
study as efficiently as possible. 

A method of determining subject size proposed by 
Bailey (1997) is based upon a variation of the binomial 
probability formula, 

1 - ( 1 - P )“ 

where , 

p = probability of the event occurring 
n = number of test subjects 

For this method, a subject matter expert or team of 
experts must first derive the likelihood, an estimate 
for p, that an element of the system will confuse any 
one test subject. For example, suppose that it has been 
determined by a panel of experts that the likelihood of 
any one test subject having difficulty identifying a 



29 



confusing air-track icon in a synthetic environment is 
0.5, that is to say using a single subject in our study 
there is a 50/50 chance of the subject having a problem. 
If two subjects are tested, the probability that the 
confusing air-track icon be identified rises to 0.75. 
This probability is calculated using the binomial 
formula presented. Furthermore, if three subjects were 
used, this probability rises to 0.87 and with seven 
subjects to 0.99. Therefore, by basing our calculations 
on the original likelihood, we can determine sample 
size. Table 3 provides a chart for determining sample 
size. Problem probability can be roughly assumed to be a 
rough estimate of problem severity (Bailey, 1997) . 



Table 3: Likelihood of Performance Test Subjects Having 

Problems 



Problem 

Probability 








Nxjinber 


of 


Test 


Subjects 






1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


.05 


. 05 


. 10 


. 14 


.19 


.23 


.26 


. 31 


.34 


.37 


.41 


.10 


.10 


.19 


.27 


.34 


.41 


.47 


. 53 


.57 


. 61 


. 65 


.15 


.15 


.28 


.39 


.48 


.56 


.62 


. 68 


.73 


.77 


.80 


.25 


.25 


. 44 


.58 


.68 


.76 


.82 


. 87 


.90 


.92 


.94 


.50 


.50 


. 75 


.87 


.94 


.97 


.98 


.99 








.75 


.75 


. 94 


.98 


.99 














.90 


.90 


. 99 



















Bailey, 1997 



After determining the number of participants and 
the length of the study, the test can be developed. Once 
the goals and concerns of the test have been defined, 
the initial tasks to test should be selected and 
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organized to best address these goals and issues. The 
tasks then need to be placed in a context that is 
understandable to the user. It is through the 
development of task scenarios that this is accomplished. 
These scenarios serve as the basis for the test. 

6. CONCLUSIONS 

Synthetic environments are an emerging technology 
that will enable individuals to perform new functions 
and accomplish older functions in a new way. Usability 
engineering and usability testing tools have recently 
been developed primarily in conjunction with standard 
graphical user interfaces. If synthetic environments are 
going to mature and become integrated into our society, 
they must be easy to use and enable individuals to 
accomplish tasks more efficiently. This maturation can 
only take place if these fields can be integrated so as 
to establish systematic and standardized methods of 
evaluation. This evolution will begin by basing 
synthetic environment designs on human-computer 
interaction principles. 

This thesis argues that the application of 
principles of human -computer interaction derived from 
existing literature and research can be applied to 
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synthetic environments. There is significant enthusiasm 
surrounding synthetic environments, but little effort 
has been made to mature the field of synthetic 
environments and utilize existing models of usability 
and user interface design. This can be best addressed by 
demonstrating what is accomplished when the methods of 
usability are applied to synthetic environments. It will 
only be through a significant development and successful 
implementation of an existing or emerging synthetic 
environment using usability principles throughout its 
design and implementation that this integration will be 
taken seriously. 
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III. 



methodology 



A. RESEARCH APPROACH 

This study involved the analysis of an existing 
decision support system and the development of 
evaluation methods based on this system. The purpose of 
this analysis was to assess the extent of the system's 
usability, to assess the effect of the interface on the 
user, and to identify any specific problems with the 
system (Dix et al, 1997) . 

B. DATA COLLECTION 

Participants . 12 participants for this study were 
recruited at the Naval Postgraduate School (NPS) in 
Monterey, California. All participants were military 
officer instructors or students at NPS and had 
previously served as Surface Warfare Officers (SWO) . Six 
of the participants had served aboard Aegis ships and 
six had not. The participants were further divided by 
experience level into one of two categories, low and 
high. Experience levels were based on a combination of 
months spent at sea and the number of deployments. The 
low experience level group on average had 44 months at 
sea and two deployments; the high experience group on 
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average had 5 8 months at sea and an average of 2.7 



deployments. Participants were distributed across four 
categories. Table 4 shows the distribution of 
participants across these categories. 



Table 4: Subject Distribution 





Low 


High 




Experience 


experience 


Non-Aegis 


3 


3 


Aegis 


3 


3 



All participants had experience as Combat Information 
Center (CIC) Watch Officers. Five participants had 
additional experience as Tactical Action Officers. Two 
of the subjects were U.S. Navy Lieutenant Commanders and 
ten were Lieutenants. 

Instrument . This study will provide a benchmark 
across usability objectives. A usability task script and 
post-task questionnaire were administered to all 
subjects. At the conclusion of the study, a post-test 
questionnaire was administered to the participants. See 
the descriptions below for specific definitions of 
obj ectives . 

• 90% Successful completion of tasks. 

• 90% Error free rate. 
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90% score of 3 or better on a 7 pt. scale (e.g., 



l=easy, 3=somewhat easy, 5=somewhat difficult, 
and 7=difficult) in ease-of-use. 

• 90% score of 3 or better on a 7 pt . scale (e.g., 

l=satisfied, 3=somewhat satisfied, 5=somewhat 
dissatisfied, and 7=dissatisf ied) in user 
satisfaction . 

Ideally, by the time a Decision Support System is 
released to the fleet, these objectives should be met 
and/or exceeded in order for the system to meet high 
ease-of-use standards. 

Procedure : Participants completed an informed 

consent form and demographic questionnaire (Appendix A 
and B) . The participants also received a usability task 
script along with a brief description of the evaluation 
scenario (Appendix C) . Participants sat directly in 
front of two 21-inch computer display monitors and 
controlled the DSS-2 with a computer mouse. The 
beginning of the usability evaluation consisted of the 
participants responding to a series of questions 
concerning their initial reaction to the DSS-2 graphical 
user interface (Appendix D) . Participants were then 
directed to read aloud and execute the tasks provided 
them in the task script. Following each series of tasks. 
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questions concerning the usability of the DSS-2 were 
presented. Additional questions concerning participant 
satisfaction as well as current understanding of the 
DSS-2 were also presented. Upon completing this phase of 
the study, participants were timed on the completion of 
tasks using the DSS-2 . The study concluded with the 
administration of a post-test questionnaire. 
Participants received no training on the DSS-2 prior to 
the usability study. The DSS-2 component names, such as 
track profile and response manager, were not used during 
interactions with the participants. These components 
were addressed in respect to the location they would be 
found on the display monitor. For example, the track 
profile component would be referred to as the area in 
the upper left side of the right monitor. 

Throughout each usability session, the following 
measurements were taken during the performance of user 
tasks. These measurements were used to assess whether or 
not each usability objective had been met. These 
measurements include: 

• Task Completion Rate: The proportion of participants 
who complete the task successfully and independently 
without critical errors. A critical error has 
occurred when the participant either requests 
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assistance from the usability engineer or commits an 
uncorrected error that results in an incorrect 
outcome for the task. 

• Error Free Rate: The proportion of participants 

completing the task without any errors, critical or 
non-critical . Non-critical errors include any error 
corrected by the test participant without 
intervention by the usability engineer or an error 
left uncorrected, but which does not affect the 
correctness of the outcome of the task. 

• User Satisfaction: The User Satisfaction rating is 

derived from a series of questions which the user 
rates on a 7 -point scale, ranging from very 
dissatisfied to very satisfied. The questions 
solicit user opinions with regard to ease-of-use, 
simplicity of the human- computer interaction, system 
functionality, and general satisfaction with the 
product . 

C. DATA ANALYSIS 

The occurrence of each of the measurements listed 
above was recorded in a spreadsheet. These data included 
any associated user-feedback information associated with 
the measurement. Frequencies of the various measurements 
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in the database were determined, both in aggregate and 
by measurement type. The categorization of participants 
by experience level and whether they had previously 
served onboard aegis ships was used in presenting the 
results. However, due to small sample size and no 
noticeable differences between categories all subsequent 
analysis was performed on all participants as a single 
group . 
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IV. USABILITY EVALUATION RESULTS 



A. BACKGROUND 



The results of this usability evaluation are 
presented in the same order they were collected. The 
participant's initial impressions of the DSS-2 graphical 
user interface are presented along with the participants 
initial impressions of the six major components. The 
DSS-2 components include the Figures 2 through 7 : 




Figure 2 : Toolbar 




Figure 3 : Geoplot 
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Figure 4 : Minicros 




Figure 5; Track Profile 




Figure 6: Response Manager 




Figure 7 : Track Summary 



40 












The participant's task completion rates, post- task 
question responses, and the answers to the post-test 
questionnaire are also provide. 

B. INITIAL IMPRESSIONS 

Overall, participant's first impressions of the DSS 
were positive (Table 5). Participants generally found 
the DSS to be a familiar interface that contained more 
information than they were accustomed to in existing 
shipboard systems. In addition, participants stated that 
their initial impression of the DSS-2 was that it aided 
situational awareness and is used to consolidate 
information. In particular, participants were familiar 
with the geoplot map display and understood that the 
toolbar would be used for manipulating the geoplot map 
(Tables 6 and 7) . Participants generally understood that 
the minicros were summaries of individual track 
information and that this information was ordered in 
some manner (Table 8) . Three participants either did not 
know what the minicros would be used for or incorrectly 
identified the meaning of the minicro functionality. 
Participants had difficulty identifying the track 
profile component of the DSS-2 (Table 9) . A majority of 
participants incorrectly believed the Track Profile 
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component to be a weapons status monitor that reflected 
the status and quantities of weapons available. The 
response manager was unfamiliar to all participants 
(Table 10). However, the concept of the response manager 
was generally understood by all participants. 
Participants stated that the response manager would deal 
with engagement orders, doctrine, rules of engagement, 
recommended actions, or a checklist. Generally 
participants correctly identified the track summary 
component, however three participants incorrectly 
assumed that the track summary information pertained to 
own-ship status (Table 11) . 
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Table 5: What is your first impression of what you see? 



• Familiar Geoplot. 

• Like it, layout. Black and white stand out. 

• Similar to JOTS display. 

• Where we are and where our battlegroup is . Focus on 
situational awareness, where we are. 

• Difficult. 

• Situational Awareness . 

• A lot of crap on the screen, cluttered, overwhelming. 

• Lot of information, used to pick out symbols. 

• Consolidated a lot of info . 

• Immediately obvious, right sight. 

• Looks cool. Intimidating. 

• Like it, used to one screen. Get more info with this. 



Table 6: What does the far left side of the left monitor 





represent to you? 


(Toolbar) 


• 


Control panel for display and tracks. 


• 


Power point . 




• 


View of situational picture. 




• 


Track contact info, select track 
decisions for you. 


symbology, make 


• 


Legend . 




• 


Filter setting. 




• 


Legend for map. 




• 


NTDS notations . 




• 


Toolbar for geoplot. 




• 


Alter geoplot. 




• 


Select what you want to look at. 
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Table 7 : Wbat does the upper left area of the left 
monitor represent to you? (Geoplot) 

• Map. 

• JOTS display, NTDS. 

• Where I am and what my radar knows . 

• Visual display of geographical picture. 

• Operating area, tactical area. 

• Geoplot. 

• Northern Persian Gulf. 

• Geoplot of surface and air contacts. 

• Threat access, radar responsiveness in a certain area. 

• Big picture, where we are. 

• Topographical map . 



Table 8: What does the bottom row on both monitors 
represent to you? (Minicros) 

• Quick Summary to what you are seeing. 

• Classification of targets, not sure how it classifies. 

• What I know of the tracks in my area. 

• Contact information, track #'s and names. 

• Weapon employment areas . 

• Contacts. 

• Information of ships . 

• Current tracks. Air and merchants set as priority time or 
threat . 

• Contact bearing range, speed, and sensor types. 

• One for each track. 

• Nothing, selection buttons? 
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Table 9: What does the top left area of the right 
monitor represent to you? (Track profile) 

• X axis questions and range. 

• What our weapons are. 

• Status of weapons systems. 

• Weapons status, a horizontal bar chart. 

• TAO stuff. Order of steps, things to do. 

• Nothing, weapons status? 

• My weapons and how much I have . 

• Our weapons . 

• How much I have, status, and range. 

• Not quite sure, our weapons status? 

• Weapons status . 



Table 10 : What does the middle-left area of the right 
monitor represent to you? (Response Manager) 

• Engagement orders . 

• When things happen, what should happen at COI/ already 
taken place. 

• Never seen anything like it. A decision matrix for ROE, 
possible defenses, weapons posture. Where we are and what 
we should do . 

• Decision-making. Decisions I need to make. 

• I don't know, maybe doctrine statement. 

• Doctrine. 

• Distance time line, envelopes. Need to do something. 

• ROE ' s . 

• Priority of actions. Recommended actions by system. 

• Don't know, a continuiim of todo's? 

• Time line. 
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Table 11: What does the top right area of the right 
monitor represent to you? (Track Summary) 

• Track details. 

• Close control, more information, emitter age. 

• Contact COI . 

• Has a tracks detailed information. 

• Specific track information. Don't know how it is 
different from (minicros) , maybe more specific. 

• All I need to know to launch. 

• Own ship, don't like presentation. 

• Ship status. 

• Contact information. 

• Blow up of minitrack status. Active/ intel /last known 
position . 

• Our own ship status. 

• Ships information and dependent on current selection. 



Once data was gathered on the participants initial 
impressions of the DSS and its components, a series of 
tasks were presented. Participants completed the tasks 
and answered post-task questions concerning ease-of-use 
and satisfaction. Task completion errors were recorded 
and task completion percentages were calculated. Task 
completion rates which were below the usability 
criterion of 90% are highlighted in Table 12 and 
discussed in Chapter five. 
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C . TASK COMPLETION 



Table 12: Task Errors and Completion Rate 



TASK 


Errors 


Completion 

Rate 


Task 1: Display the track numbers of all contacts in 

the map display. 


4 


67% 


Task 2: Locate and select track number 7012. 


0 


100% 


Task 3 : Change the map to display the directions all 

tracks are moving. 


1 


92% 


Task 4: Remove all unknown tracks from the map 

display . 


0 


100% 


Task 5: Display all surface unknown tracks. 


3 


75% 


Task 6: Display all tracks. 


0 


100% 


Task 7; Change the size of the map to better see the 
tracks displayed. 


3 


75% 


Task 8: Identify the range of map sizes that are 

available . 


12 


91 


Task 9 r Explore these two buttons . When 
exploring, select map size of 128 nm. 


finished 


0 


100% 




'T— IfW 




Task 10: Please point to own-ship on the map display. 


1 


92% 


Task 11: Identify which tracks are potential threats. 


6 


50% 


Task 12: Determine whether track 7013 is within its 

weapons release range. 


3 


7 5% 


Task 13: Determine whether track 7013 is within own- 

ship's weapons envelope. 


1 


92% 


Task 14: Identify the bearing and range of track 

7016 . 


0 


100% 


Task 15: Check and see what the most recent warning 

information is regarding track 7011. 


3 


75% 


Task 16: Identify the most recent warning information 
for track 7011. 


2 


83% 


Task 17: Select track 7016. 


0 


100% 


Task 18: Locate own-ship position on the right 

monitor . 


2 


S3,% 


Task 19: Using information available on the right 
monitor, determine whether track 7016 is within its 
weapons release range. 


3 


75% 


Task 20: Using information available on the right 
monitor, determine whether track 7016 is within own- 
ship's 5/54 guns weapons envelope. 


0 


100% 


Task 21: Select track 7017. 


0 


100% 


Task 22: Click on verify airspace, issue a level 1, 

and change CIWS to auto/ready. 


1 


92% 


Task 23: Select track 7012. 


0 


100% 
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Figures 8 through 26 show participant responses to 
each of the post-task questions. Each figure shows 
either ease-of-use or satisfaction ratings for all 12 
participants . The bars in the figures represent 
individual participant ratings. These bars are grouped 
according to the category of user and the bars within 
each grouping are order according to relative experience 
levels within the group. Tables 13 through 43 summarize 
participant comments and what participants thought could 
make the completion of the task easier. 
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Figure 8: How easy/difficult was it to display the track 

numbers? (Question 1) 

Table 13 : Comments & what would make this easier? 

(Question 1) 

• Had to look for it. Training. 

• Clutters display, lose visual reference of contacts. 

• Just a little more time to study screen. 

• Very easy considering 1st time use. Change buttons to 
read easier (e.g.) "Display all track #'s". 

• Shift click or mousedrag over all contacts. 

• Different color background to make it standout add to the 
"show" title for instance "Display on Map". 

• I was not sure what I was doing was going to work the 
first time. Experience with the interface. 

• Took a step and had to decide if it was a collective 
action or serial. Pre-knowledge. 

• Easy once I knew what to do. 

• Advanced training, change label to "show all tracks" or 
something like that. Experience. 



49 



7 -I 




Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 

Participants 



Figure 9: How easy/difficult was it to find track number 

7012? (Question 2) 



Table 14: Comments & what would make this easier? 

(Question 2) 



• 


Put track #'s in order on bottom display. 




• 


What will happen to display w/in 
contacts . 


traffic zones. 


>20 


• 


No problem. 






• 


Need a 10 key to enter track # . 






• 


Experience with the interface. 






• 


Track number hides symbol. If the 


tracks exceed 


the 




display. Area, may need a summary 
categorical . 


list that is 




• 


If I understood the ordering of the track #'s at 


the 




bottom. 
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Figure 10: How easy/difficult was it to read the track 
number on the map display? (Question 3) 

Table 15: Comments & what would make this easier? 

(Question 3) 

Possibility of having #'s separate. This has its 
disadvantages too! 

On large scale chart hard to distinguish individual #'s 
Make larger . 

Looking for track # highlighted. Arrange tabs by other 
' sorts ' . 

Flashing contact # or something to make it stand out - gets 
larger for example. 

How far from screen will the user be? More than one user 
may have to use a display. 
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Figure 11: How easy/difficult was it to display the 
course leaders in the map display? (Question 4) 



Table 16: Comments & what would make this easier? 

(Question 4) 

• Used history function first. 

• Display all buttons. 

• Experience. 

• Experience. 
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Figure 12: How easy/difficult was it to remove unknown 
tracks from the map display? (Question 5) 



Table 17: Comments & what would make this easier? 

(Question 5) 



• 


Had to click to determine if on/off. 




• 


Did it individually by air, sfc, sub, not in one step. 


• 


Little time needed to understand there were 


toggle 




buttons . 




• 


Very easy since I had the experience of the 


previous 




task. 




• 


Experience . 
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Figure 13: How easy/difficult was it to display all 
surface unknown tracks on the map display? (Question 6) 



Table 18: 



Comments & what would make this easier? 
(Question 6) 



Symbology is a plus. 
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Participants 

Figure 14: How easy/di£f icult: was it to understand when 
the track type buttons were selected? (Question 7) 



Table 19: Comments & what would make this easier? 

(Question 7) 

• Training. 

• Noise was good indication, but won't work in shipboard 
environment. Distinct color change would be better, (e.g. 
black vs. white) . 
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Question 8 : How would you explain what this 



does ? 




Table 20: Comments & what would make this easier? 

(Question 8) 

• Zooms by factor 2 . 

• Goes down to next pre-selected button and increases by 
1/2 the current range. 

• Decreases range scale. 

• Decreases map size, smaller range scale. 

• "+" = Zoom In. 

• Zoom in. 

• Zoom in . 

• Zooms out, lowers scale. 

• Zooms in range on increment per click. 

• Increase zoom. 

• Zooms in range on increment per click. 

• Zoom in. 



Question 9 . 



How would you explain what this 




does ? 



Table 21: Comments & what would make this easier? 

(Question 9) 

• Unzooms by factor 2. 

• Decreases by 2 the current range. 

• Increases range scale. 

• Increases map size, larger range scale. 

• ”-"=Zoom Out. 

• Zoom out . 

• Zoom out . 

• Zooms in. 

• Zooms out range on increment per slide. 

• Decrease zoom, increase size of area covered. 

• Zooms out . 

• Zooms out . 
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Question 10. Does the positioning of these buttons 

(plus on the left / minus on the right) meet your 

expectations? Five participants responded yes, seven 

subjects said no. 

Table 22 : Comments & what would make this easier? 

(Question 10) 

• Nijmbers decrease to the left, therefore sign should be to 
the left. 

• Expect minus on left associate small w/left, large 
w/right . 

• At first I had them reversed. 

• "+" on the Right? 

• I expected the noirmal to the left though I saw after 

trial that it means to increase the scale. For me it was 
a compatibility error. 

• Would help to have a drag zoom to center and zoom. 

• Seems like "+" should be on the right, on the left. 

• Prefer to be able to select my own choice of 
magnification. 
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Figure 15: How easy/difficult was it to identify own- 
ship on the map display? (Question 11) 

Table 23: Comments & what would make this easier? 

(Question 11) 

• Standard NTDS symbology used. 

• Hook self. 

• Provide legend in Track types for own ship or the 
different nodes show that this is a geo-center display. 

• I assumed that the viewer could always choose the center 
of the screen wherever he wants . 
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Participants 

Figure 16: How easy/difficult was it to identify whch 
tracks were potential threats? (Question 12) 

Table 24: Comments & what would make this easier? 

(Question 12) 

• Standard NTDS Threats are red, unknown white, etc. 

• Color on model is yellow/expected NTDS standard of red. 
Either use NTDS colors and symbols or more time and 
familiarity with model. 

• Color is deceiving, I was drawn to colored icons, not all 
threats a TGT that had not had any additional evaluation, 
would not be colored, and could be missed. What is a 
threat in this scenario, can I change that criteria and 
then display them? 

• This is ambiguous at best. In Gulf all tracks are 

potential threats. 
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Figure 17 : How easy/dif f icult was it to determine 

whether track 7013 was within its weapons release range? 

(Question 13) 

Table 25: Comments & what would make this easier? 

(Question 13) 

• Comparing to left screen made it clearer. 

• Multiple envelopes are going to be hard to see. 

• This information is only based on an assumed 
configuration, may lead to wrong decision making. 

• Not sure what the red and gray grids mean. This is really 
just a training issue, but easily learned. 

• Experience with the interface. 

• Pre knowledge of red versus white would make it easier. 

• Difficult to figure it out the first time. Rename button 

to say "weapons ranges " . 
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Figure 18: How easy/difficult was it to determine 
whether track 7013 was within own-ship's weapons release 

range? (Question 14) 



Table 26: Comments & what would make this easier? 

(Question 14) 



• 


Comparing to left screen made it clearer. 








• 


Multiple envelopes are going to be hard to 


see . 






• 


This is known info, easy to decide. 








• 


This is really just a training issue but easily 


learned. 


• 


Because I was guided to the weapons button 


on the 


left 




screen . 








• 


Pre knowledge of red versus white would make it 


easier . 


• 


Difficult to figure it out the first time. 


Rename 


button 




to say "weapons ranges". 








• 


Experience . 
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Figure 19: How easy/dif f icult was it to identify the 
bearing and range of track 7016? (Question 15) 



Table 27 : Comments & what would make this easier? 

(Question 15) 

• Familiarity. 

• Bottom tab not so clear. Bearing =??, Range=??. 

• I need compass display, not for this, but relative 
positions for ship's head for weapons envelopes. 

• Not readily recognizable, but it is only due to 
familiarity with the display. 

• Titles for course and range displayed. 

• Unsure whether 312/16.6 represents range/brg or cse/sro. 
Label the names . 
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Question 16 . Do you think there is any meaning to 
the ordering of these items? Eleven participants said 
yes, one said no. 

Table 28: If yes, what do you think the ordering means? 

(Question 16) 

• Range and threat . 

• Threat . 

• Threat level. Unsure of ordering within categories 
(Threat, Unk, Nuet) no apparent categorization by 
platform. Appears to list closest contact first. 

• Organized from higher potential threats to lower 
potential threats. 

• Yes, excellent quick reference for track data. Ordering 
according to threat? 

• Don't always know what is driving the ordering. 

• Higher threats on left. 

• Threat priority. 

• Potential threat order. 

• I think they are ordered in terms of threat. 

• Ordered by threat /unk/ friend and then by contact order. 
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Figure 20: How easy/difficult was it to identify the 
most recent warning information for track 7011? 

(Question 17) 

Table 29: Comments & what would make this easier? 

(Question 17) 

• Familiarity and knowing "alerts" had to be clicked on the 
symbol . 

• Training . 

• Unfamiliar with pull down menu. Not require continued 
user interface with mouse to view alert menu. 

• My Understanding of Question? Button to read "Warnings" 

• Track updated, I did not see and called wrong 
information! ! ! 

• What are those numbers, if its time in needs to be in 
00:00:00 format. 

• Easily found and understood. 

• Except I had to hold the mouse button down to view 
alerts. Menu should stay up after clicked. 

• Pre knowledge. 

• Didn't know where to find it. Maybe a flashing warning 
light . 

> Experience. 
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Figure 21: How easy/difficult was it for you to select 
the alert button and view the alert window? (Question 

18) 



Table 30: Comments & what would make this easier? 

(Question 18) 

• Familiarity. 

• Unfamiliar with pull down menu. Not require continued 
user interface with mouse to view alert menu. 

• Didn ' t catch my eye right away . 

• Couple times to learn to hold down. 

• Kind of small. Keep alert up, let any button close 
window . 

• Easily found and understood. 

• Except I had to hold the mouse button down to view 
alerts. Menu should stay up after clicked. 

• I would like it to stay in view when c 1 i eked . 
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Question 19: Click and hold the alert button. What 

do the numbers on the right mean to you? Seven 

participants believe the number on the right 

corresponded to the time elapsed since warning occurred 

and one the time the warning occurred. 

Table 31: Comments & what would make this easier? 

(Question 19) 

• Blue is significant, gray contact updates. 

• Lt. Blue means increase threat issue. 

• White: general info. Blue: ROE specific info. 

• Green appears to be a more serious alert. 

• Blue=Warning or threat, Grey=neutral info. 

• Blue is associated attack/defending. 

• Color means new alert, use red. 

• Blue means what affect if has on me. Grey is general 
inf ozTnation . 

• Contact within weapons range, I would expect that the 
colors would change (maybe to red) as the contact gets 
closer. 

• White general, blue warning, red? Perhaps hostile action. 

• Blue means high significance. 

• White -narrative. Blue-threat. 
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Figure 22; How easy/di££icult was it to understand what 
the numbers in the alert window mean? (Question 20) 



Table 32 : Comments & what would make this easier? 

(Question 20) 



• 


I just didn't know when I saw them, time since 
new to me . 


report 


is 


• 


Training . 






• 


Still unsure. Familiarity with system or standard use 


of 




Zulu time on all areas. 






• 


Column header = Time Elapsed. 






• 


Both actual and elapsed time on target. 






• 


00:00:00 format. 






• 


Use a plus symbol next to the time to indicate 


how long 




since time zero. 






• 


No indication of what they mean. Labeling. 
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Figure 23: How easy/difficult was it to understand what 
the colors in the alert window mean? (Question 21) 

Table 33: Comments & what would make this easier? 

(Question 21) 

• I am still not sure. Familiarity, if I used the system 
the colors would be easy to remember. 

• Add a feature for contacts weapons release range in RED. 

• Need to find pattern. Use more than 2 color if it is 
going to colorize. 

• Use red. 

• I had to think about it . 

• Yellow is a better warning color, blue is too passive. 

• No indication of what they mean. Labeling. 
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Figure 24: How easy/difficult was it to determine 
whether track 7016 was within its weapons release range 
using the infromation displayed in the right monitor? 

(Question 22) 



Table 34: Comments & what would make this easier? 

(Question 22) 



• 


Familiarity. 






• 


Training. 






• 


I need a representation giving 


weapon ' s masking 


areas 




unless I know the graphs account for it. 




• 


History is less important than 


current velocity 


for 




weapons . 






• 


I'm not sure how to interpret 


the graph. Training. 
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Figure 25: How easy/dlfflcult was it to determine 

whether track 7016 was within own-ship's weapons release 
range using the information displayed in the right 
monitor? (Question 23) 

Table 35: Comments & what would make this easier? 

(Question 23) 

• Training-but by now I have some . 

• Have available weapons as a pull down menu. Click 
weapons, release mouse, see choices, make choice by 
clicking desired weapon. 

• Select and drag is hard, select/select . 

• Easily understood. 

• Make weapons window a pop down and stay vice hold down 
and find. 

• I'm not sure how to interpret the graph. Training . 
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Table 36: What do you expect the ASPECT button to do? 

(Question 24) 

• Show angle on bow for TGT and TGT angle 

• Show highlighted COI target angle, heading, altitude, etc 

• I don ' t know 

• 2=D shift by 90 degrees or 180 degrees. 3-D view 
possibly? 

• TGT aspect 

• Target w/relation to ship's head. 

• Change from center on me to a center on him. 

• Place threat contact at (0,0) axis. 

• Weapons release envelope of current velocity of contact. 

• I don ' t know 

• Target aspect . 

Question 25: The aspect button only met one 

participants expectations, ten said that it did not. 

Table 37 : Comments & what would make this easier? 

(Question 25) 

• It tells own ship aspect to TGT, which is easy, and now I 
expected target angle. 

• It's a good tool, but w/ "aspect", I expected to see "nose 
on" "crossing", etc. 

• It give whether your weapons engagement criteria are met 
or not. 

• Took me a few seconds to understand what it was telling 
me. Also, the weapons button is hard to use. 

• What ' s it mean? 

• Needs to be a lot bigger, should be scalable like map, 
should be primary weapons display. 

• No reason, I'm just off. 

• I did think that it would do what it did until I tried 
it, and then it made sense. 

• It is a weapons engagement aspect for own ships weapons. 

• Because I didn't know what it was for. 

• Does not give me target aspect, provides weapons I can 

use against target with my present heading . 
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Figure 26: How satisfied/dissatisfied are you with the 
ASPECT button feature? (Question 26) 

Table 38: Comments & what would make this easier? 

(Question 26) 

• I like it! 

• Satisfied with the information it is giving me, not with 
its ease-of-use. 

• Excellent! Rename button to "Arc of Coverage". 

• Bad button name, it is showing weapon cut outs, we all 
worry where we can not shoot and how to maneuver to 
shoot. Change color if in and out, edge discrimination. 

• Once I understood what it meant, this feature is 
excellent . 

• Needs to be in a bigger window and maybe a different 
name . 

• Should be bigger. 

• Not what I expected. 
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Table 39: What would you expect to do with this area 
based on the track selected? (Question 27) 

• Use it for reports and ROE 

• Go down the check list to see what’s been done 

• Time line for deconf liction, verification of COI, weapons 
status upgrade, countermeasures and reports. Basically 
how to classify or fight this target according to ROE. 

• Interact/warning 

• Line coincides with doctrine packages (I.e. follow these 
doctrine) 

• Checklist. 

• Conduct doctrine based query factors. 

• Perform the actions that the vertical line is in. 

• I think that I can click on what warnings I have given 
and the display will change colors to remind me later. 

• Respond to recommendations. 

• Do the action when the line reaches the left side of the 
bubble . 

• Actions can take at this range. 



Table 40: What did clicking in the Response Manager mean 

to you? (Question 28) 

• Orders were given to do the three task clicked 

• Only that the task has been completed by someone else 

• Actions have been taken. 

• Issued track a warning, took a self-defense measure with 
CIWS, checked for territorial airspace. 

• Told CIC that were in level 1 & switch to CIWS auto . 

• It has been done. 

• Should be being carried out . 

• I am verifying that my airspace is clear, issuing a level 
1 warning to the contact and setting CIWS into auto. 

• Memoiry aid not a command to change CIWS status a display 
of current status . 

• Click: issue order to appropriate station. 

• No idea . 

• When within range the selected actions will take place. 
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Table 41: What does the greyed-out area, change CIWS to 
auto /ready mean to you? (Question 29) 

• Already ordered. 

• Only that the TAO thinks it ' s done . 

• DSS is recoinmending this course of action. 

• Means the system is standing by to fire on a potential 
threat . 

• That doctrine is being followed -active. 

• That is a ship system, it is already in that state or 
been clicked. 

• Means it is already complete w/regard to 7013. 

• The CIWS will automatically track and the system is 
operational . 

• Means that my CIWS status is auto/ready. 

• The CIWS is still in auto/ready. 

• It's been accomplished. 

• Places CIWS in auto/ready. 



Table 42: What Is the relationship between the Track 
Profile and the Response Manager? (Question 31) 

• They are on same scale 

• Range of contact, engagability vs. defensive steps taken 
by own ship 

• Range of COI is the same top shows physical engagement 
limits bottom shows ROE and weapons, posture 
recommendations . 

• One displays possible decisions to make on a track and 
the other provides a visible aspect of that decision. 

• Top-gives threat envelopes and the track bottom gives 
what doctrine to follow according to the track. 

• Show points in relative distance that require /suggest or 
move points of action. 

• Distance axis. 

• Spatial in that as the contact further progresses in the 
envelope certain tasks must be completed. 

• They show that a contact is within my weapons range and 
tracks the ROE that I have followed. 

• Range of contact to ship. Range of Recommended actions. 

• They are covered by range of target to own ship. 
Represents the actions that need to be taken. 

• Both display range to target ship. 
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Table 43: What does the Response Manager white line mean 

to you? (Question 32) 



• 


Actual range. 




• 


Range to own ship . 




• 


COI range . 




• 


Range of contact to ship, also gives 


an indication of 




where you should be in your decision 


making process. 


• 


The doctrine to follow according to 


that specific track. 


• 


Present distance. 




• 


Distance target is from ship. 




• 


Distance line. 




• 


Decreasing range of contact. 




• 


Range of contact in question. 




• 


Range to target ship. 





Figure 27 shows the post-task rating average for 
each of the participants. For example, the bar farthest 
to the left of Figure 27 shows that the average rating 
given across all post-task questions by this participant 
to be 3.0. Figure 28 shows the average post-task rating 
for each post-task question. For example, the bottom bar 
representing question one indicates that the average 
rating for this question was approximately 2.5. 
Questions for which the rating exceeded the usability 
criterion of 3.0 include questions: 

• 13 - How easy/difficult was it to determine 

whether track 7013 was within its weapons release 
range . 

• 17 - How easy/di f f icult was it to identify the 

most recent warning information for track 7011? 

• 20 - How easy/diff icult was it to understand what 
the numbers in the alert window mean? 

• 21 - How easy/difficult was it to understand what 
the colors in the alert window mean? 
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Questions 




Figure 27 : Average Participant Ratings with Standard 

Deviation 



26 { 




Ratings 

(1-easy, 3>somewhat easy, S-somewhat difficult, 7>difficult) 

Figure 28: Average Question Rating with Standard 

Deviation. 
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Immediately following the usability evaluation 
participants were given a post-test questionnaire. 
Figures 29 through 42 show participant ratings for ease- 
of-use and satisfaction for the DSS and its' components. 
All ratings were within the established usability 
criteria with the exception of the Track Profile. The 
Track Profile component received an average ease-of-use 
rating of 3.2 and participant satisfaction rating of 
3.2. 
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Lower Non-Aegs Upper NorvAegs Lower Aegis Upper Aegis 



Participants 



Figure 29: How easy/difficult was it for you to use the 

Geoplot? (Question 1) 

Table 44: How could the Geoplot be improved? 

• Own ship speed leader . 

• Switch +/- on range scales. 

• Good geoplot, detailed digital maps would be excellent. 

• Declutter button like the one on the JMCIS. Add symbology 
to the plot. 

• Zoom area option. Center ship option. Speed leader on own 
ship . 

• Be able to choose center instead of always own-ship. 




Partiapants 

Figure 30: How satisified/dissatisf ied were you with the 

Geoplot? (Question 2) 
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Lower Non-Aegis Upper Norv>Aegis Lower Aegis Upper Aegis 

Participants 



Figure 31: How easy/dif f icult was it for you to use the 

Minicros? (Question 3) 

Table 45: How could the Minicros be improved? 

• Limit information on Minicro so more can be displayed. 

• Hold training on capabilities. 

• Use labels for course, speed, range, altitude, and 
associated units. 

• Did the bearing and range represent CPA data or the 
contacts bearing and range? 

• Threats need more markers, anything w/closing CPA needs 
to be marked in some manner. 

• What happens if list is too much? 

• Labeling of Range/Dist or CSE/Speed. 

• Display track course and speed. 




Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 

Participants 



Figure 32: How satisfied/dissatisfied were you with the 

Minicros? (Question 4) 
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Partictpants 

Figure 33; How easy/dif f icult was it for you to use the 
Track Proflile? (Question 5) 

• Training. 

• Make own ships weapons a pull-down menu w/o continuous 
user control of mouse. 

• Aspect and weapon buttons were a little difficult to 
understand at first. 

• The "aspect" picture seems to be the better one to use. 
Suggest switching the two pictures. 

• Need weapons envelopes on top down like aspect. Needs to 
be primary interface . 

• Remove history, add speed leader. Use different color for 
history and speed leader. 

• Unsure how to interpret. Training. 

• Too many choices. All weapons on 1 display. 
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Lower Non-Aegis Upper NorvAegis Lower Aegis Upper Aegis 

Participants 



Figure 34: How satisified/dissatlsf led were you with the 

Track Profile? (Question 6) 
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Lower Non-Aegis L^per Non-Aegis Lower Aegis Upper Aegis 

Participants 



Figure 35; How easy/difficult was it for you to use the 

Toolbar? (Question 7) 

Table 46; How could the Toolbar be improved? 

• Complete NTDS symbology. 

• Needed training. 

• Make threat color red . 

• + and - range buttons . 

• Change clicks into distinct visible color changes. 

• Add a select all, remove the sound when selecting. 

• Be able to combine background elements . 

• Make it so I can choose my own magnification. 




Participants 

Figure 36; How satisified/dissatisfied were you with the 

Toolbar? (Question 8) 
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Participants 

Figure 37: How easy/difficult was it for you to use the 
Response Manager? (Question 9) 

• Tell me if these are directives or recommendations. 

• Excellent doctrine tool. 

• I think feedback on time action complete could be added 
to each line, so TAO knows it's done. 

• If distance line passes into the area, have the area 
color stand out until task is completed. 

• Unsure what it is used for. Training. 




Lowr Noo-Ae9s Upper Non-Aegs Lower Aegis Upper Aegis 

Participants 



Figure 38: How satisified/dissatisfied were you with the 
Response Manager? (Question 10) 
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Lower Non-Aegis Upper Non-Aegis Lower Aegis Upper Aegis 

Participants 



Figure 39: How easy/difficult was it for you to use the 
Track Summary? (Question 11) 

• Training. 

• Allow user to manually type in track # . Need to add time 
to CPA and/or time to COI at weapons release range. 

• CPA was a little difficult to catch at first. 

• I had trouble finding the CPA. 

• Track selection is hard to discover needs marker, so you 
can tell it's a button. 

• Add for keyboard numerical pad change of contact. Further 
explain CPA by adding CPA bearing and CPA range lines . 

• Clicking and holding on the track # to display 
info/select a new track did not seem natural to me. 

• Increase size of own ship CRS and SPD. Add own ship speed 
leader . 
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Lower NoivAegis Upper NorvAegis Lower Aegis Upper Aegis 

Participants 



Figure 40: How satisfied/dissatisfied were you with the 
Track Summary? (Question 12} 
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Lower NorvAegis Upper NorvAegis Lower Aegis Upper Aegis 



ParticiparTts 

Figure 41: Overall, how easy/difficult was it for you to 

use the DSS? (Question 13) 



• Training . 

• Pull-down menus for alerts and own ships weapons. 

• Training, more use of color or shape as indicators, 
rather than sound . 

• Very high learning curve. Vast improvement to tactical 
display and management. 

• Ensure track number does not block symbology. 

3.5-] 




Lower NorvAegis Upper NorvAegis Lower Aegis Mppe^ Aegis 

Participants 



Figure 42; Overall, how satisfied/dissatisfied were you 

with the DSS? (Question 14) 
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Table 47 : What: changes would you like to make to the 

DSS? (Question 15) 



• Faster response to "weapon" (upper right) and have an 
alert to the right of minicro that says "more contacts". 

• Can you modify the Response Manager for changing ROE, 
optasks, or missions such as amphibious assault or 
maritime interdiction ops? 

• + and - on range scales reversed. 

• Colors are "eye-catchers" and even have meanings. Use 
very descriptive words. Change track profile picture to 
the "aspect" picture that was clearer to me. Place track 
data in table format (this may save space and be easier 
to look at) . 

• Change weapon envelopes into geographic aspect display to 
account for weapon's masking. 

• Allow keypad input for the contact number in Track 
Summary. Add titles to CPA BRNG and CPA RNGE and add an 
estimated time to CPA. Add a plus or minus to the alert 
times, not how long after the alert was posted. 

• Slight interface change. For example, have the menu stay 
up after you click the alerts for a contact. 



Table 48: What were the worst aspects o£ the DSS 
interface and why? (Question 16) 

• Split screen. 

• No time to CPA. No time to COI w/in its weapons release 
range or w/in your weapons release range. 

• Increase the size of the "CPA" indicator. Weapon pull- 
down menu is difficult to use. 

• Track profile picture is still a little confusing. Keep 
things simple. 

• No idea about some features because they give no 
indications they are buttons. 

• Too much reliance on the mouse. With 2 screens and 
looking in other areas besides the 2 monitors, the mouse 
arrow is easily lost. 

• Weapons button was difficult to find and understand. 

• History in weapons display, aspect display. 
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Table 49: What were the best aspects of the interface 

and why? (Question 17) 



• Ease-of-use. 

• Good data, lots of info displayed here. 

• Easy to see threats and weapons envelopes . 

• The decision making process time line with target range. 

• Use of colors. All on "1" screen. It was all right there, 

• Geographic w/map display is nice. 

• Minicro-I like the quick and pertinent information on the 
contact. Response Manager-excellent doctrine aide. 
Aspect-allows for quick maneuvering recommendations to 
OOD. 

• Overall I think it was an easy to use interface, 

• Liked having so much information simultaneously visible. 



Table 50: Where there any parts of the interface that 
you found confusing or difficult to understand? 

(Question 18) 

• Multiple threat displays possibly pop-up, sizeable 
windows . 

• No, not really. 

• Not really, difficulties primarily due to unfamiliarity 
w/system. 

• It took me a few minutes to understand aspect. 

• Track profile picture - it showed weapon engagement 
envelopes but I'm still a little confused by it. The 
"aspect" picture was clearer to me. I'm not sure what the 
track profile picture axes were? 

• Time late on the alert box needs to be 00:00:00 format. 

• Adding new contact as far as reporting one that is not on 
the minicro. It is easy to get used to the display and 
not look at the geoplot . 

• Weapons button. Track summary. 

• Track profile area. 
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V. 



SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 



A. SUMMARY 

The DSS did not meet these usability objectives of 
90% across all tasks for task completion. The overall 
task completion rate across all tasks was 84%, 6% below 

the usability criterion initially established. When 
examined by individual task, on 13 tasks the DSS 
surpassed the 90% usability criterion level and on 10 
tasks the DSS-2 did not meet the usability objective. 
The only task on which all participants committed an 
error was the identification of the map's upper and 
lower range scale. Half of the participants were unable 
to properly identify potential threats and one-third of 
the participants had difficulty displaying all the track 
numbers on the map display. A quarter of all 

participants committed errors on five different tasks. 
These tasks were : 

• Display all surface unknown tracks; 

• Change the size of the map to better see the 
tracks displayed; 

• Check what the most recent warning information is 
regarding track 7011; 

• Determine whether track 7013 is within its 
weapons release range; 

• Determine whether track 7016 was within its 
weapons release range using the track profile 
component . 
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Two participants incorrectly identified the most recent 
warning information for track 7011. 

Overall, the DSS-2 did not meet the 90% usability 
objective across all tasks for ease-of-use. Eighty- two 
percent (8% below the 90% objective) of the ease-of-use 
questions averaged a rating of somewhat easy or better. 
The four questions that did not meet the ease-of-use 
usability objective goals were: 

• How easy/dif f icult was it to determine whether 
track 7013 was within its weapons release range? 

• How easy /diff icult was it to identify the most 
recent warning information for track 7011? 

• How easy /diff icult was it to understand what the 
numbers in the alert window mean? 

• How easy/difficult was it to understand what the 
colors in the alert window mean? 

Four ease-of-use questions received a rating of six or 

higher by at least on participant, where a rating of 

five corresponded to somewhat difficult and a rating of 

seven corresponded to difficult . These questions were: 

• How easy/difficult was it to display all tracks? 

• How easy/difficult was it to identify whether track 
7013 was within its weapons release range? 

• How easy/difficult was it to understand what the 
numbers in the alert window mean? 

• How easy/difficult was it to determine whether 
track 7016 was within it's weapons release range 
using the information displayed in the right 
monitor? 
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The usability objective for satisfaction was met, 100% 
of satisfaction ratings were below a rating somewhat 
satisfied . 

During the usability study, participants were 
questioned about the functionality and user interface 
design of some components of the DSS-2 . The following 
information was compiled from participant' responses. 

The range scale feature presented some difficulty 
to participants. Participants were able to correctly 
explain how the range magnification buttons functioned 
after exploring the button. However, seven participants 
stated that they believed that the labeling of the 
buttons was reversed. 

When questioned about the ordering of the minicros, 
all but one participant believed the minicros were 
ordered in some manner. Of the eleven participants who 
believed the minicros were ordered, nine correctly 
assumed that they were ordered by threat. When asked to 
select a track that did not have a corresponding 
minicro, four of the 12 participants committed an error. 
A majority of participants expressed surprise that some 
tracks did not have a corresponding minicro. 

All 12 participants had difficulty selecting the 
alert button on the minicro. In addition, although 
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eleven participants correctly identified the numbers 
found in the alert window, every participant expressed 
frustration that these numbers were not labeled. Four 
participants stated that additional or different formats 
for these numbers might be more appropriate. In general, 
participants expressed significant dislike for the way 
the alert buttons had to be selected. The interaction of 
having to click and hold the button to read the 
information contained within the pop-up window caused 
frustration. Many participants felt that this 
interaction would be inappropriate during shipboard 
operations . 

The aspect button surprised almost all 
participants. Ten participants stated that this button 
did not meet their expectations, however, nearly all 
participants liked the feature. The predominate 
criticism of the aspect button was its size, many 
participants expressed that it was too small. 

The post-test questionnaire examined the 
participants overall experience with the DSS-2 and with 
each individual component. Overall, results of the post- 



test questionnaire 


suggest 


that the 


DSS-2 met 


the 


usability 


ob j active 


criteria 


initially 


established 


of 


somewha t 


easy or 


better 


for ease-of-use 


and 
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satisfaction. The average DSS-2 ease-of-use rating was 
2.6 and the average satisfaction rating was 2.3. The 
only DSS-2 component that did not meet the ease-of-use 
and satisfaction usability objectives in the post-test 
questionnaire was the track profile component. This 
component received a 3.2 for ease-of-use and 
satisfaction. All participants reported that they 
enjoyed working with the DSS-2 and believed that it 
would be a significant step forward in CIC information 
management . 



B . RECOMMENDATIONS 

Table 51 details the usability issues encountered 
by study participants during this usability evaluation 
and provides corresponding recommendations. 



Table 51: DSS Component Issues and Recommendations 



COMPONENT 


ISSUE 


RECOMMENDATION 


Geoplot 


Track symbology 

1. Some participants did not like 
the coloring of potential 
threats and unknown tracks . 
Participants stated that 
unknowns were potential 
threats . 


1. Conduct further 
research to 
determine the 
optimal use of 
track color. 


Toolbar 


Track Numbers 

1. Some participants wanted the 
track number button's name to 
be changed to "Display all 
tracks # ' s " . 


1 . Change existing 
button label. 
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Range Magnification 
1. All participants had 

difficulty discovering the 
upper and lower bounds of the 
range scale . 



1 . Remove button 
option and 
replace with a 
slider widget. 



2 . Many participants chose the 
incorrect range magnification 
button to resize the geoplot. 



2 . Change the range 
magnification 
icons relative 
positions . 



3 . Some participants wanted to be 
able to enter any range using 
a numerical keypad. 



3 . Provide numerical 
keypad capability 
to enter a 
specific range 
magnification . 



Minicro 



4 . One participant wanted to be 
able to use the mouse on the 
geoplot to select an area to 
be magnified by clicking and 
dragging across the area. 

General 

1. Many participants wanted 
abbreviated labels on the 
information displayed. 



4 . 



1 . 



Provide this 
dynamic map range 
selection 
capability . 



Provide more 
labeling within 
the alert window. 



2 . Several participants were 
surprised when the selected 
minicro changed. A few users 
looked at another area of the 
DSS-2 and did not expect the 
selection to change when the 
priority did. 



2 . Maintain the 

selected state of 
the minicro, even 
if a priority 
changes . 

POTENTIAL 
DANGEROUS . 



Alert Information 
1 . Many participants expressed 
frustration when they could 
not bring up the alert 
quickly. Participants clicked 
on the alert button and didn't 
realize they had to click and 
hold it to bring up alert 
window . 



1. Redesign the 

alert selection 
mechanism so 
that one click 
will open alert 
window and a 
second click 
will close it. 



2 . Many participants wanted the 
information contained in the 
alert window to be labeled. 



2 . Label 

information . 



3 . Some participants wanted the 
time of the alert, as well as 
the time since the alert, to 
be displayed. 



3 . Test the 

feasibility of 
having both 
options . 



4 . Many participants did not like 
the light blue color and 
preferred for red or yellow. 



4 . Additional 
research on 
color . 
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Track 


Weapons Envelope 






Profile 


1. Some participants had 


1 . 


Provide training 




difficulty identifying which 




and easily 




weapons envelope, red or blue. 




accessible help 




related to the selected track. 




feature . 




2. One participant wanted the 


2 . 


Provide the 




ability to have multiple 




capability to 




weapons envelopes displayed 




have multiple 




simultaneously. 




weapons 
envelopes . 




3 . One participant had difficulty 


3 . 


Training . 




identifying the location of 
own-ship . 








Aspect Insert 

1. Many participants stated that 


1 . 


Redesign and test 




the aspect button should be 




alternative 


- 


labeled "Weapons Cut-Out" or 
"Cut-Out" . 




labeling . 




2 . Many participants wanted the 


2 . 


Increase size of 




Aspect Insert to be larger. 




Aspect Insert. 




3 . Some participants preferred 


3 . 


Training . 




the Aspect Insert to the Track 
Profile . 






Response 


General 






Manager 


1. Many participants stated that 


1 . 


Training . 




they were unsure whether the 
options in the Response 
Manager were recommendations 
or requirements. 








2 . Many participants wanted to be 


2 . 


Existing feature 




able to edit the Response 




that was not 




Manager . 




tested . 


Track 


General 






Summary 


1. Some participants had 


1 . 


Redesign and test 




difficulty reading the 




alternative 




information contained in the 




colors to 




Track Summary due to poor 




facilitate 




background and text coloring. 




reading . 



C. CONCLUSIONS 



This study examined the usability of the 

systematic manner and established where 

product is today against usability 
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DSS-2 in a 
the DSS-2 
criteria . 



Furthermore, participants identified usability issues 
that can be addressed in future design and research 
efforts. Based on the information obtained from this 
evaluation, recommendations to address the usability 
issues were made. Overall, in terms of ease-of-use and 
satisfaction, participants reacted positively to the 
DSS-2. Participants felt the DSS-2 would assist them in 
maintaining situational awareness and was a tool that 
would be useful onboard Navy ships. 

The methodology applied in this study was useful in 
the evaluation of the DSS-2. This study demonstrated 
that traditional human-computer interface usability 
methods could be directly applied the evaluation of 
synthetic environments. The DSS-2 is a simple synthetic 
environment represented on two computer monitors. Given 
the success of this methodology with the DSS-2, it would 
be appropriate to use this methodology in evaluating 
more complex synthetic environments . 
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APPENDIX A: CONSENT FORM 
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CONSENT FORM 



Usability Evaluation of the Decision Support System 

Principal Investigator: LT Dylan Schmorrow 

Operations Research Department 
Naval Postgraduate School 
Monterey, CA 93943 

I, , consent to my participation in the research project titled 

Usability Evaluation of the Decision Support System. 

I understand that I am free to withdraw my participation in the research at any time and that if I do 
I will not be subjected to any penalty or discriminatory treatment. 

I have been given the opportunity to ask questions about the research and received satisfactory 
answers. 

I understand that any information or personal details gathered in the course of this research about 
me are confidential and that neither my name nor any other identifying information will be used or 
published without my written permission. 

I understand that if I have any complaints or concerns about this research I can contact: 

George Conner 

Operations Research Department 
408-656-3306 



Signed by: 



Date 
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APPENDIX B; DEMOGRAPHIC INFORMATION FORM 
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DEMOGRAPHIC INFORMATION 



Name: Rank: Designator: 

Years of service: Time in rank (months): Curriculum: 

Previous Command: Primary billet: 

Months onboard: Months in shipyard (if applicable): 

Months standing CIC watch: Months standing TAO watch: 

Hours spent on computer each week: Operating system primarily used: 



Qualifications: 


Circle One 


QIC Watch Officer? 


Yes 


No 


SWOS Department Head Course? 


Yes 


No 


SWC Qualified? 


Yes 


No 


STWO Qualified? 


Yes 


No 


TAO Qualified? 


Yes 


No 


Aegis Qualified? 
Other? 


Yes 


No 



Experience Summary -- List assignments relevant to SWO experience 
(include Combat Systems) 



Command 



Months 



Deployments Frequency 

LANT 

PAC 

MED 

PERGULF 

Others 
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APPENDIX C: 



TASK SCRIPT 
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Background 



This project was spawned by the 1988 USS Vincennes 
incident where an Aegis cruiser engaged in a littoral 
warfare peace-keeping mission shot down an Iranian 
Airbus. Investigations following the incident suggested 
that stress may have affects on decision making, and 
that these effects were not well understood. This 
project was established to address these concerns. 

This prototype Decision Support System (DSS) was 
developed to enhance Navy tactical decision making based 
on "naturalistic" decision processes. Displays were 
developed to support critical decision making tasks by 
Naval watch officers operating in a shipboard Combat 
Information Center. 

SCENARIO 

This scenario has the ship operating independently 
in the northern Persian Gulf 50 nm to the east of Kuwait 
City. You are on a presence patrol and have been 
directed to remain within 5 nm of your current position 
to demonstrate US resolve. Weapons and warning status: 
Yellow and Tight. At scenario start you are on course 
020, speed 7 knots. Visibility is reduced to 
approximately 4 to 5 nm in dust and haze. Local time is 



1100 . 
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Task 1 



Display the track mimbers of all contacts in the map 
display . 



Task 2 

Locate and select track number 7012. 
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Post Task Questions 



1. How easy/difficult was it to display all the track 
mimbers? 



1 2 
easy 



3 4 

somewhat 
easy 



5 

somewhat 

difficult 



6 7 

difficult 



Comments : 



What might have made this task easier? 



2. How easy/difficult was it to find track number 7012? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 
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3 . How easy/difficult was it to read the track number on 
the map display? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



Task 3 

Change the map to display the directions all tracks are moving. 



Task 4 

Remove all unknown tracks from the map display. 



Task 5 

Display all surface unknown tracks. 



Task 6 

Display all tracks. 
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Post Task Questions 



4. How easy/difficult was it to display the course leaders 
in the map display? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



5. How easy/difficult was it to remove unknown tracks from 
the map display? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



6. How easy/difficult was it to display all surface unknown 
tracks on the map display? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



7. How easy/difficult was it to understand when the track 
type buttons (i.e., surface, unknowns) were selected. 



1 2 
easy 



3 4 

somewhat 
easy 



5 

somewhat 

difficult 



6 7 

difficult 



Comments ; 



What might have made this task easier? 



Task 7 



Change the size of the map to better see the tracks 
displayed. 



Task 8 

Identify the range of map sizes that are available. 
Read aloud the upper and lower range levels. 



Task 9 

Explore these two buttons. When finished exploring, select 
map size of 128 nm. 
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Post Task Questions 



8 . How would you explain what this button does 




”5 



9. How' would you explain what this button does 




? 



10. Does the positioning of these buttons (plus on the 
left / minus on the right, meet your expectations? 

Yes / No 



If no, why not?. 
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Task 10 



Please point to own-ship on the map display. 



Task 11 

Identify which tracks are potential threats? 



Task 12 

Determine whether track 7013 is within its weapons release 
range . 



Task 13 

Determine whether track 7013 is within own-ship's weapons 
envelope . 
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Post Task Questions 



11. How easy/difficult was it to identify own-ship on the 
map display. 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



12. How easy/difficult was it to identify which tracks 
were potential threats? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



13 . How easy/difficult was it to determine whether track 
7013 was within its weapons release range? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



14. How easy/difficult was it to determine whether track 
7013 was within own-ship's weapons release range? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier?. 



Ill 



Task 14 



Identify and read aloud the bearing and range of track 
7016 . 



112 



Post Task Questions 



15. How easy/difficult was it to identify the bearing and 
range of track 7016? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 
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041 ♦/34 i 


Surface j 


^ 3000 '—' 
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^ 3000 


C4bst«r II 1 


' ' ..Pr mas-40 . 


l>c-eea-1224 


APS-115 


II* ES 


n« wr 


IFF ;; 


fto BT 


No HT 


Mo IFF 


Utertsj! 




falMrtsj 




_lPi. 



16. Do you think there is any meaning to the ordering of 
these items? 

YES / NO 

If yes, what do you think the ordering means? 
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Task 15 



The DSS is continually updating information on the behavior 
of all tracks. Check and see what the most recent warning 
information is regarding track 7011. 



Task 16 

Identify and read aloud the most recent warning information 
for track 7011. 
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Post Task Questions 



17. How easy/difficult was it to identify the most recent 
warning information for track 7011? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

What might have made this task easier?. 



18. How easy/difficult was it for you to select the alert 
button and view the alert window? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

What might have made this task easier?. 



19. Click and hold one of the alert buttons. 

A. What do the numbers on the right mean to you? 

Time elapsed since warning occurred. 

Time warning occurred. 

Other, please described 

B. What do the colors mean to you? 
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20. How easy/difficult was it to understand what the 
numbers in the alert window mean? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier?. 



21. How easy/difficult was it to understand what the 
colors in the alert window mean? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 
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Task 17 



Select track 7016. 



Task 18 

Locate own-ship symbology on the right monitor. 



TASK 19 

Using information available on the right monitor, determine 
whether track 7016 is within its weapons release range. 



TASK 20 

Using information available on the right monitor, determine 
whether track 7016 is within own-ship's 5/54 Guns weapons 
envelope . 
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Post Task Questions 



22. How easy/difficult was it to determine whether track 
7016 was within its weapons release range using the 
information displayed in the right monitor? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 



23 . How easy/difficult was it to determine whether track 
7016 was within own-ship's weapons release range using 
the information displayed in the right monitor? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 



What might have made this task easier? 
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24. What do you expect the ASPECT button to do? 



25. Click it. Does this meet your expectations? 
Yes /no 

If no, why not? 



26. How satisfied/dissatisfied are you with the ASPECT 
button feature? 



1 

satisfied 



2 



3 

somewhat 

satisfied 



4 



5 

somewhat 

dissatisfied 



6 



7 

dissatisfied 



Comments : 
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Post Task Questions 



Task 21 



Select track 7017. 



Question 
based on 




27 : What would you expect to do with this area 
the track selected? 



Varfftt sirspoce 
1 St waniiDfl 

C^ws U juto/rsai^ Mif systems ^ 

I 2 ntf^orni«g _ 



Exec«t« EW poclagss 



Jrt 



wwmtag 



Report 



to sMlor 



Stlngor Datscimioa} 



Aim 



Doploy Docoga 



Flro Florei/Worntofl Shol 



Eatogo / Do Ml EfMOQo iiocuioo 



Task 22 

Question 28: Click on verify airspace, issue a level 1, and 
change CIWS to auto/ready. What does it mean to you? 



Task 23 

Select track 7013 

Question 29: Please describe what the dark gray area, "CIWS 
to auto/ready", means to you. 



Did you expect the "CIWS to auto ready" to be dark gray? 
YES / NO 

Are the actions listed here REQUIRED or RECOMMENDED? 
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4001 (OwnShip) 



Cover 



Execute Vih paciCQaes 



n weming 



liiuminfite 






Report to senior 



Alert Stinger Detodiment 



Deploti Decoys 



Fire Flores/Womlnj Shot 



Enyoye / Do not Eniioge decision 



Question 31: What is the relationship between the two areas 
above? 



Question 32; What does the white line represent to you? 
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APPENDIX D; POST-TEST QUESTIONS 
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OVERALL POST-TEST QUESTIONS 




Figure 43: Geo-Plot with Desaturated Map and Variable 

Coded Symbology 



1. How easy/difficult was it for you to use the Geo-Plot? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved? 



2. How satisfied/dissatisfied were you with the Geo-Plot? 



1 

satisfied 



somewhat 

satisfied 



5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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Figure 44: Sample Minl-CRO 



3 . How easy/difficult was it for you to use the Minicros? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments ; 

How could this area be improved? 



4. How satisfied/dissatisfied were you with the Minicros? 



1 2 
satisfied 



3 4 

somewhat 
satisfied 



5 6 7 

somewhat dissatisfied 
dissatisfied 



Comments ; 
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5. How easy/difficult was it for you to use the Track 
Profile? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved? 



6. How satisfied/dissatisfied were you with the Track 
Profile? 



1 

satisfied 



somewhat 

satisfied 



5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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- Background - - — - 
Topolog g I 




7 . How easy/difficult was it for you to use this area? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved?. 



8. How satisfied/dissatisfied were you with this area? 

I I I I I I I 

1 2 3 4 5 



satisfied 



somewhat 

satisfied 



5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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Figure 46: Response Manager 



9. How easy/difficult was it for you to use the Response 
Manager? 



1.2 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved? 



10. How satisfied/dissatisfied were you with the Response 
Manager? 



1 2 
satisfied 



3 4 

somewhat 
satisfied 



5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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Figure 47: Track Summary 



11. How easy/difficult was it for you to use the Track 
Summary? 



12 3 4 

easy somewhat 

easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved?. 



12 . How satisfied/dissatisfied were you with the Track 
Summary? 



1 

satisfied 



2 



3 4 

somewhat 
satisfied 



5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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Figure 48: TADMUS DSS Integrated Display 



13. Overall, how easy/difficult was it for you to use the 
DSS? 



1 2 
easy 



3 4 

somewhat 
easy 



5 6 7 

somewhat difficult 

difficult 



Comments : 

How could this area be improved? 



14. Overall, how satisfied/dissatisfied were you with the 
DSS? 



1 2 
satisfied 



3 

somewhat 

satisfied 



4 5 6 7 

somewhat dissatisfied 

dissatisfied 



Comments : 
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Question 15: If you could improve the DSS system, what 
changes would you like to make? 



Question 16: Overall what were the worst aspects of the DSS 
interface and why? 



Question 17 : Overall what were the best aspects of the interface 
and why? Please describe. 



Question 18: Where there any parts of the interface that 
you found confusing or difficult to understand. Please 
Describe. Please describe. 
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